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ABSTRACT 



This chapter presents the results for the Multi-Agency Study 
of Teacher Enhancement Programs, a study of professional development programs 
believed to represent best practices in professional development for science 
teachers . Working through an interagency government task force coordinated by 
the National Science Foundation (NSF) , representatives from six groups, the 
Departments of Energy, Education, Health and Human Services, the National 
Aeronautics and Space Administration, NSF, and the Smithsonian Institution 
conducted the study. The evaluation was designed to document the teaching 
practices promoted in the selected programs and to examine the impacts of the 
programs in terms of: (1) effects on classroom practice; (2) differences in 

effects on classroom practice by type of program; (3) effects on teacher 
leadership and dissemination of findings; and (4) the importance of supports 
at the district and school levels. Programs selected were expected to meet 
most of the following criteria: (1) an instructional approach that emphasizes 

hands -on/minds -on activities; (2) a standards -based approach that aligns 
curriculum, instruction, and assessment with local, state, and national 
standards or frameworks; (3) development activities that extend over time, 
including follow-up when participants return to their schools; and (4) direct 
involvement of participants with the scientific process. Additional 
considerations governing selection were stability, inclusion of teachers who 
are themselves from traditionally underrepresented populations, and inclusion 
of programs that were carried out within a systemic reform context. 

Thirty- four programs were included in the study, which was divided into three 
phases: programs documentation and description, a mail survey of all 
participants, and site visits to 13 of the initial 34 programs. Twenty-eight 
of these programs were characterized as development programs; the remaining 
six as research programs. Findings confirm the value of well-designed 
professional development experiences. (Contains 11 references.) (MVL) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



ED 465 588 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

□ This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



° Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 




Findings from the Multi- 
Agency Study of Teacher 
Enhancement Programs 



Joy Frechtling 
Westat, Inc. 

, Conrad Katzenmeyer 
National Science Foundation 



This chapter* presents the results for the Multi-Agency Study of Teacher 
Enhancement Programs, a study of professional development programs 
believed to represent best practices in professional development for 
science teachers. Working through an interagency government task force 
coordinated by the National Science Foundation (NSF), representatives 
from six groups, the Departments of Energy, Education, Health and Human 
Services, the National Aeronautics and Space Administration, NSF, and the 
Smithsonian Institution conducted the study. The evaluation was designed 
to document the teaching practices promoted in the selected programs and 
to examine the impacts of the programs in terms of: 1) effects on classroom 
practice, 2) differences in effects on classroom practice by type of program, 

3) effects on teacher leadership and dissemination of findings, and 4) the 
importance of supports at the district and school levels. Programs selected 
were expected to meet most of the following criteria: 1) an instructional 
approach that emphasizes hands-on/minds-on activities; 2) a standards- 
based approach that aligns curriculum, instruction, and assessment with 
local, state, and national standards or frameworks; 3) development activities 
that extend over time, including followup when participants return to 
their schools; and 4) direct involvement of participants with the scientific 
process. Additional considerations governing selection were stability, 
inclusion of teachers who are themselves from traditionally underrepresented 
populations, and inclusion of programs that were carried out within a 
systemic reform context Thirty four programs were included in the 
study, which was divided into three phases: program documentation and 
description, a mail survey of all participants, and site visits to 13 of the 
initial 34 programs. Twenty-eight of these programs were characterized as 
development programs; the remaining six as research programs. Findings 
confirm the value of well-designed professional development experiences. 

‘This summary of evaluation findings is an update of the document. Best practice 

in action: Final report of the multi-agency study of teacher enhancement programs , 

Frechtling (1997) published by Westat, Rockville, MD. 
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Not only were they seen by participants to be personally and professionally 
valuable, but also the data show that a teacher’s participation has important 
effects on what happens in the classroom. Second, in the short term, 
programs that are tied directly to what is desired in the classroom have 
the most immediate payoff. Third, the school or district context, and the 
support provided for the application of learning make a big difference in 
the extent to which changes actually occur. Finally, the linkage between 
participation in professional development activities and the promotion of 
teacher leadership and dissemination of benefits is weak. Without explicit 
supports and direction for such undertakings, they are not likely to occur in 
any consistent or meaningful way. 

The basic goal of professional development programs is to broaden 
and deepen the disciplinary and pedagogical knowledge of teachers. 
Attention to issues of professional development has increased since the 
beginning of this decade, when the President and 50 state Governors 
included improved professional development for teachers as one of 
eight national education goals: 

By the year 2000 , the Nations teaching force will have 
access to programs for the continued improvement of 
their professional skills and the opportunity to acquire the 
knowledge and skills needed to instruct and prepare all 
American students for the next century \ (National Education 
Goals Panel, 1993) 

Professional development programs in science and mathematics 
education have been the subject of widespread interest in recent years. 
This emphasis reflects, in part, the heightened importance of teacher 
education to systemic educational reform. Essentially, systemic reform 
calls for coordinated changes in all parts of the educational system to 
ensure that new and higher standards of achievement in science and 
mathematics are met by all students, regardless of gender or ethnic and 
linguistic background. Educators agree that the fundamental changes 
required to produce both academic excellence and equity rest largely 
on the continued growth and professional development of the Nation’s 
teaching force. Reflecting that understanding, each year thousands of 
teachers take advantage of the opportunities for professional growth 
offered through a variety of means ranging from limited teacher 
workshops during the school year to summer institutes and multi-year 
programs of study. 

These activities, generally known as “professional development 
programs,” share one or more of the following goals: 
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• Broadening teacher knowledge and skills; 



• Helping teachers apply new knowledge and skills in the 
classroom; 



• Increasing teachers’ awareness of hands-on and laboratory 
materials and developing strategies for use in the classroom; 
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understanding of careers in mathematics 
and science and the skills needed for them; and 



• Fostering teacher leadership. 

Within this general framework, two types of more comprehensive 
programs are generally distinguished. 

Development programs are multidisciplinary offerings typic- 
ally held in classroom settings at local colleges, regional 
education service centers, county or central/district offices 
of education, or laboratories. Normally these programs run 
from 2 to 4 weeks and include presentations, demonstrations, 
lectures, hands-on activities, cooperative group work, field 
work, and time for discussion, planning, and reflection. 
These programs usually focus on encouraging a “hands-on/ 
minds-on” instructional approach and have a primary goal of 
changing curriculum, instruction, and assessment. 



Research programs are similar in many respects, but they 
tend to be more intense and focused on immersing teachers 
in the scientific process by carrying out research projects in 
a university or federal laboratory setting under the guidance 
of working scientists. As initially developed, these programs 
were intended to engender a deep understanding of the 
research process in teachers and increase their knowledge 
about emergent content areas. Transfer to teaching and 
the classroom was seen as one byproduct of this increased 
understanding. 

Both types of professional development programs may be a single 
year or multi-year experience and generally include followup sessions 
during the school year. Both also may provide additional materials or 
resources for participants to use in their classrooms. 

Despite their importance to educational reform, professional 
development programs, including these more indepth, comprehensive 
experiences, have until quite recently been the subject of relatively 
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little comprehensive evaluation. Further, most of the studies reported 
in the literature have relied solely on teachers’ self-reports. The 
majority of these evaluations looked at participant satisfaction with 
the programs and examined reported impacts on teacher renewal and 
enthusiasm for teaching. A much smaller number also looked at 
application of knowledge to the classroom. These studies typically 
concluded that most participants feel satisfied with the programs they 
attend, enjoy the opportunity to network with their peers, use what 
they have learned once back in their classrooms, share what they have 
learned with others, and believe that their new teaching behaviors 
have beneficial effects on their students (Frechtling, Sharp, Carey, & 
Vaden-Kieman, 1995). 

While such reports are encouraging, direct evidence of changes in 
classroom practice and resulting improvements in student performance 
have been sorely lacking. Understanding this situation and recognizing 
the problems it posed, the Dissemination and Evaluation Working 
Group (DEWG) — a government body operating under the National 
Science and Technology Council — coordinated an effort by federal 
agencies that support professional development projects to undertake 
an extensive evaluation of their efforts. 2 As a result, a multifaceted 
study was initiated in 1 994. This chapter presents the results of that 
study. 



Study Overview 

Working through DEWG, six agencies initiated a study of 
professional development programs: the Departments of Energy 
(DOE), Education (ED), and Health and Human Services (HHS), the 
National Aeronautics and Space Administration (NASA), the National 
Science Foundation (NSF), and the Smithsonian Institution. The 
work was coordinated by NSF and carried out by three independent 
research firms — Westat, SRI International, and the National Center for 
Improving Science Education. In the long term, the evaluation and 
other efforts of the DEWG were designed to meet two basic goals: 

• Identifying those government-supported professional 
development programs in science, mathematics, and 
technology that are most effectively implementing and 
encouraging “best practice,” and 



2 The DEWG was an interagency group whose mission was to enhance and expand 
the evaluation and dissemination of mathematics and science programs. 
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• Assessing the extent to which these programs are contributing 
to the national effort to improve science education in the 
schools. 

More immediately, this evaluation was designed to document the 
teaching practices promoted in the selected professional development 
programs believed to represent best practice in science education at 
the time of study initiation. The study examined the impact of these 
programs on teachers and their teaching, assessed the extent to which 
dissemination took place, and determined participant satisfaction with 
the programs and their outcomes. In addition, the study tried to 
identify the contextual factors that affect a teacher’s ability to apply 
new learning and use new approaches in the classroom setting. 

Based on an extensive review of the professional development 
literature available in 1994, best practice was defined in terms of four 
elements: 

• An instructional approach that emphasizes hands-on/minds-on 
activities; 

• A standards-based approach that aligns curriculum, instruction, 
and assessment with local, state, or national standards or 
frameworks; 3 

• Development activities that extend over time, including 
followup when participants return to their schools; and 

• Direct involvement of participants in the scientific process. 

Using these and other selection criteria, program officers from 
each of the six agencies nominated professional development programs 
that they considered successful in delivering professional development 
in science teaching. Additional considerations governing selection 
were 1) stability 4 ; 2) inclusion of teachers who themselves were from 
populations traditionally underrepresented in science or who work 
with significant numbers of students from those underrepresented 
groups; and 3) inclusion of programs that were carried out within a 
systemic reform context. 



3 See National Council of Teachers of Mathematics (NCTM), National Research 
Council (NRC), and Project 206 l’s Benchmarks (Listed in References at the end 
of this chapter) for descriptions of standards-based instruction and professional 
development. 

4 It is recommended that programs nominated be in at least their second or third year 
of operation to avoid first-year, startup problems. 
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Thirty-four programs identified through this process were selected 
for the evaluation. Twenty-eight of the programs were characterized 
as development programs and the remaining six as research programs. 
The evaluation of these combined professional development efforts 
was carried out in two phases, summarized in Exhibit 1 . During Phase 
I (summer 1994), evaluators made site visits to the 34 professional 
development programs while they were in progress, documenting 
program events and practices, reviewing program materials, and 
interviewing participants and other key players. The purpose of these 
visits was to document the extent to which programs exhibit the 
characteristics of best practice. 

During Phase II, the evaluators looked at the impacts of 
participation on teachers when they returned to their classrooms. Two 
complementary techniques were used: a survey of both 1993 and 1994 
program participants from each of the 34 programs; and more indepth 
analyses (called “case stories”) of participants from 13 programs; 
these programs were selected because of variation in their design 
and implementation. The survey was used to get a broad picture of 
teachers’ perceptions of the programs and their impacts. The case 
stories were designed to gather data that would corroborate (or refute) 
the findings from the surveys, as well as to provide a more indepth 
picture of program results in a limited number of cases. Detailed 
reports of the separate components of this study have been presented 
in a series of reports issued between November 1995 and March 
1997. 5 

The Impacts of Professional Development Programs 

The multi-agency study provided confirmation of a number of 
previous findings regarding professional development programs. First, 
the participants in these programs are not entirely representative of 
the U.S. science teaching force. While they resemble the overall 
teaching force in terms of age, number of years teaching, gender, 
racial/ethnic distribution, and major field of study, as a group they 
have a higher preponderance of advanced degrees and are more likely 
to have participated previously in extended professional development 



5 Phase I methodology and findings is presented in Ruskus and Luczak (1995). The 
findings from the site visits are presented in Westat, Natonal Center for Improving 
Science Education, and SRI International (1996). The findings from the survey are 
presented in Carey and Frechtling (1997). The present report summarizes some of 
these study findings. 
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Exhibit 1. Study design 



Best practices evaluation of professional development 
(TE) programs 


Phase I. Program 
Documentation and 
Description 


Phase II. Participant Followup 


( Summer 1994 ) 


( 1 994-95 School Year) 


Site visits 


Mail (census) survey 


Site visits 


34 TE programs 


All 1993 and 1994 


1-2 teachers from 


exemplifying 


participants from 34 


13 of the Phase I 


best practice 


Phase I programs 


programs 


• Modeling of best 


• Changes in 


• Evidence of 


teaching 


teaching 


best practices 


practices 


behaviors 


transferred 


• Immersion in 


• Participant 


from the 


“doing science” 


satisfaction 


program 


• Standards-based 


• Feelings of 


• Objective 


professional 


renewal 


validation of 


development 

• Followup 
activities 


• Leadership 
roles/knowledge 
dissemination 

• District/school 
effects on other 
measures 

. 


self-report 
data from 
survey 



programs than the national population (Tables 1, 2, and 3). Thus, 
the findings reported here, as in other programs that primarily serve 
volunteer populations, must be interpreted with that caveat in mind. 

Second, the study affirmed the previous reports that from a 
personal point of view, participation in summer institutes and followup 
activities is seen as very valuable and energizing. The following 
comments from participants in teacher development programs (Carey 
& Frechtling, 1997) were typical: 
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Table 1 . Percent distributions of teacher development participants, 
by teacher characteristics and by grade level (n = 1,481) 



Teacher Characteristics 


All 

Participants 


Grade Level 

Elementary Middle Secondary 
1-6 7-8 9-12 


Whole sample 


100% 


32% 


31% 


37% 


Gender 


Female 


65 


85 


27 


41 


Male 


35 


15 


73 


59 


Minority status 


White 


83 


84 


78 


87 


All other races 


17 


16 


22 


13 


Advanced degree 


Yes 


59 


52 


60 


65 


No 


41 


48 


40 


35 


Major field of study 


Education 


59 


87 


70 


26 


Science or math 


33 


6 


19 


69 


Other 


7 


7 


11 


5 


Number of years teaching 


5 or less 


25 


23 


16 


15 


6 to 19 


52 


54 


53 


52 


20 or more 


22 


22 


31 


33 


Administrative position held 
in school 


Department chair 


18 


21 


14 


20 


Curriculum coordinator 


10 


11 


13 


6 


Other administrator 


17 


11 


15 


23 


None 


55 


56 


58 


51 


Attended other science 


teacher enhancement 










programs of 40 hours or 


more in the past 3 years 
Yes 


47 


39 


47 


55 


No 


53 


69 


53 


45 



Note. Because of rounding, percents may not add to 100. 
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Table 2. Percent distributions of research participants, by teacher, 
school, and target science class characteristics (n=l 16) 



Background characteristic Percent of teachers 



Teacher characteristics 
Gender 

Female 31 

Male 69 

Minority status 

White 87 

All other races 13 

Advanced degree 

Yes 82 

No is 

Major field of study 

Education 15 

Science or math 80 

Other 3 

Administrative position held in school 

Yes 45 

No 55 

Attended other science teacher enhancement programs 

Yes 55 

No 45 

School characteristics 
Metropolitan status 

Central city 30 

Urban fringe 32 

Town 18 

Rural 20 

Percent minority enrollment 

Less than 10 52 

1 1 to 50 28 

More than 50 20 

School reform in science education 

Yes 58 

No 42 

Target class characteristics 
Instructional level 

Elementary (1-6) 0 

Middle ( 7 - 8 ) 15 

Secondary ( 9 - 12 ) 85 

Ability level of students 

Low ability 2 

Average ability 23 

High ability 34 

Mixed ability * 41 



Note. Because of rounding, percents may not add to 100. 
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Table 3. Population statistics for U.S. science teachers 



Grade Level 



Teacher characteristic 


Elementary 

(1-6) 


Middle 

(7-8) 


Secondary 

(9-12) 


Gender 


P^rrva Jp* 


ai m 

y ivo 


69% 


34% 


Male.. 


9 


31 


66 


Minority status 


White 


88 


89 


95 


All other races 


12 


12 


5 


Master’s degree 


Yes 


34 


42 


57 


No 


66 


58 


43 


Major field of study 


Science 


2 


17 


63 


Science education 


0 


2 


6 


Other education 


86 


63 


22 


Other fields 


12 


18 


10 


Number of years teaching 


5 or less 


23 


23 


21 


6-19 


58 


53 


44 


20 or more 


19 


25 


35 


Hours of professional development 
in last 3 years* 


None 


43 


23 


21 


Less than 16 


51 


53 


55 


16-35.: 


4 


16 


13 


More than 35 


3 


9 


11 



*Weiss and colleagues report professional development by hours. Also, 
elementary grades for these data are 1*4, and middle grades are 5-8. 

Note: Because of rounding, percents may not add to 100. 

Source: J. Weiss, M. Matti, and P. Smith, Report of the 1993 National 
Survey of Science and Mathematics Education, Chapel Hill, NC: Horizon 
Research, Inc., 1994. 
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Rejuvenation of the desire to teach great science! Ideas, 
stories, resources and support from everyone resulted in a 
feeling of power and joy in doing science in. the classroom. 
Confidence in knowing what I wanted to do was right and it 
would work. 

I returned to my school so motivated about teaching that other 
teachers were aware of my “glow. ” It was the greatest mood 
lifter I have had in a very long time. 

In addition, the study documented some other outcomes that help us 
better understand the more far-reaching benefits that well designed 
programs can provide. These findings are explored in more detail 
below. 

Effects on Classroom Practices 

The central question driving the multi-agency study was whether 
or not evidence can be found that the benefits of professional 
development programs extend to the classroom and the delivery of 
instruction. Based on the combined results of the case stories and the 
surveys, it is our conclusion that the answer is a cautious “yes.” 

We found considerable evidence that teachers were applying what 
they had learned about standards-based instruction in science to their 
classrooms. And, although not all teachers achieved the same level 
of competency, all groups showed progress and provided evidence of 
improved teaching. Gains were found for participants in both teacher 
development and teacher researcher programs. 

Instructional impacts were explored through a variety of means. 
In the case stories, a fundamental question addressed in the classroom 
observations and interviews was whether or not teachers were using 
hands-on, inquiry-based methods and whether classroom activities 
reflected the content, instrumentation, or techniques introduced in the 
summer programs. The surveys queried teachers about their teaching 
practices and the frequency with which sfudents 1 engaged in various 
kinds of activities. Teachers were asked to report oh the use of 
selected teaching strategies both prior to and after their summer 
experiences. 

Taken together, the findings show considerable use of many of 
the elements of best practice and suggest that teachers’ professional 
development experiences both reinforced and introduced standards- 
based practices that ultimately were translated into classroom 
instruction. For example, a participant in one of the teacher 
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development programs who was observed using hands-on techniques 
that were partially inquiry-based stated: 

/ have always used hands-on experiences with my students, 
even before [this program dealing with museum experience]. 6 
The program helped to give me more creative ideas and 
experiences to use in my science classes. ...I guess Museum in 
the Field did not change my teaching style, but enhanced ii. 7 

At another site the observer drew the following conclusions about 
Susan, a participant in one of the teacher development institutes, 
focused on space science: 

At Susan ’s school, the infusion of space science from K-6 is 
a direct outcome of the materials that the program provides. 

The entire elementary science program at this school is 
built around the ideas of inquiry, exploration, and hands-on 
experience of the phenomena imder study... Susan clearly 
demonstrated hands-on inquiry-based teaching throughout 
the 90-minute lesson. 

Finally, corroboration of impact was frequently found in interviews 
with supervisors and colleagues. Again, in another case story, the 
following was reported: 

The principal believes that... “he [the participating teacher] 
would have left education without this experience. I’ve 
seen [teaching] changes this year. He has more confidence 
that what he ’s presenting is relevant to students. He stresses 
relevance in whatever he does. He ’s more hands-on than any 
other science teacher, and this is how it should be... ” 

This is not to say that all programs prepared teachers to 
successfully translate what they learned to classroom activities. For 
example, our research program, rated very valuable in terms of 
participants’ own acquisition of new skills and knowledge, failed the 
translation test. 

Steve indicated that he could not use much of the content 
from the program in his classes... the subject matter is far too 
esoteric for any high school classroom application. 

6 In referring to programs, we use the pseudonyms assigned in Ruskus and Luczak, 
1995. 

’Programmatic examples are taken from Westat, National Center for Improving 
Science Education, and SRI International, 1996. 
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i Survey data (Carey & Frechtling, 1997) support the conclusion 

i that program participation impacts classroom instruction. At the same 
! time, however, the data indicate that not all aspects of standards- 
based instruction were realized with equal strength. For example, 

; while 59 percent of the participants in teacher development programs 
1 indicated that they were using new hands-on research projects and 
50 percent reported using new explanations or examples in their 
teaching, only 37 percent indicated that they had made changes in 
their use of alternative assessments. 

Figure 1 shows the reported changes in teacher practices for 
six specially selected pairs of behavior among teachers attending 
teacher development programs. One member of each pair represents 
more traditional teaching practices; while the second represents more 
current views about best practices. As the exhibit shows, while 
change in the desired direction occurred for each of the pairs of 
practices, gains were not uniform, and levels of implementation 
differed substantially after program participation 

An interesting contrast can be found at pairs a and c. Although 
the reported prevalence of the two practice choices is roughly the 
same in a and c prior to the program participation, the amount of 
change toward use of best practice is substantially different. Pair a 
contrasted use of cooperative learning with student independent work. 

! There, the use of cooperative learning increased from 33 percent of 
: the participants reporting substantial use to 74 percent selecting this 

response. The corresponding decrease in student independent work 
went from 37 percent to 6 percent reporting frequent usage. Pair 
c contrasted indepth study of selected topics with comprehensive 
coverage of topics at the expense of detail. Indepth coverage increased 
froni 27 percent prior to participation to 48 percent after participation. 
The corresponding figures for comprehensive coverage were 36 
percent to 22 percent. It is easy to see that while both of these changes 
would be considered positive, the reported adoption of indepth study 
changes far less and reaches a much lower end level than that of 
1 cooperative learning. 8 



j relative| y more limited attention to indepth study of topics is confirmed by data 

from studies such as the Third International Mathematics and Science Study, which 
indicates that U.S. teachers adopt indepth coverage far less than their international 
| counterparts. 
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Prior to program At this time 





Figure 1. 

Percentage of participants reporting change in use of various teaching 
techniques after participation in teacher development programs* 
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Prior to program At this time 





*Exhibit contrasts those reporting high usage — 4 or 5 on a 5-point 
scale — with those reporting low usage— 1 or 2 on a 5-point scale. 
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Differences were also found when the data for the teacher 
researcher programs were examined (Figure 2). Again, while change 
was in the right direction for all pairs of practices examined, changes 
were uneven and differed sharply in some cases from the teacher 
development programs. 

The study also found some important differences in extent of 
classroom application when teacher characteristics were considered. 9 
Characteristics associated with fuller translation to the classroom 
included 

• initial level of use of standards-based techniques — those 
who already used standards-based instruction before program 
participation were more likely to show more use of these 
practices after program participation; 

• school level — teachers who were at the elementary/middle 
level were more likely to report using instruction reflective of 
best practices than were high school teachers; and 

• autonomy — teachers who had a greater sense of control over 
their environments reported greater changes than those who 
felt they had less control. 

However, one of the most important findings was that there was 
no difference in the benefits received by teachers from different 
racial/ethnic groups or teachers who taught different types of 
students. Although the data from the case stories suggest that certain 
environments pose greater challenges than others, and in absolute 
terms standards-based instruction was found to be less prevalent in 
the central city and urban fringe schools than in the suburbs, the 
benefits of professional development were in evidence across the 
board. Thus, it appears to be important to encourage teachers working 
with urban populations to participate as fully as possible in high- 
quality professional development programs. Clearly, policymakers 
would be well advised to consider ways of providing incentives for 
such teachers, and for the programs that serve them. 



’These analyses of differential effectiveness for different types of teachers were only 
carried out for those participating in the teacher development programs. Because of 
the relatively smaller sample of participants in the researcher programs, these more 
fine-grained analyses could not be carried out. This issue is considered further in the 
next section on characteristics of the most effective programs. 
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Effects of Program Characteristics 

A second question of central interest was whether or not we could 
identify programs that appeared to be most effective in promoting 
change in classroom practices and, if we could, what the characteristics 
of the more effective programs might be. The results of the study 
indicated that teacher development programs that provided experience 
with the “model science classroom” had the most immediate impact 
on classroom practice. For the teachers in the development programs, 
experiences related to research and the research process showed 
relatively less classroom impact. 

What does it mean to provide experience with the model science 
classroom? Although there is no one way to do this, the general 
notion is that these programs provided direct experience in using and 
developing the kinds of instructional practices that it is hoped the 
teachers will eventually incorporate into their day-to-day teaching. 

This contrasts with programs that placed greater emphasis on the 
scientific experience, on doing, learning about, or observing scientific 
research. Characteristics of these more effective programs include 

• participating in hands-on activities for use in the classroom, 

• planning how information could be used in the classroom, 

• developing curriculum units, 

• engaging in challenging problem solving, 

• collaborating with scientists or other staff, and 

• interacting with program participants. 

While the results for the teacher development programs were 
relatively clear, the picture for the teacher researcher programs was 
not. Because of sample size limitations, we do not know from this 
study what the relative impacts of well-implemented teacher researcher 
programs might be compared to other approaches, and the potential 
benefits of the research experience require further examination. 

We did find, however, that the two types of professional 
development programs attracted different types of teachers. Participant 
differences can be classified into three general categories: personal, 
school, and classroom characteristics. 

1 . Personal characteristics: Relative to participants in teacher 

development programs, those in research programs 
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Figure 2. 

Percentage of participants reporting change in use of various teaching 
techniques after participating in research programs* 
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• were more likely to be male (69 versus 32 percent), and 

• more often had degrees above a bachelor’s (82 versus 57 
percent). 

2. School characteristics: Participants in research programs taught 

in schools that were more often 

• secondary (85 versus 32), and 

• private (14 versus 5 percent). 

3. Classroom characteristics: Participants in research programs 

taught science classes that were more likely to 

• meet for 5 or more hours per week (65 versus 37 percent), 
and 

• be composed of higher ability students (34 versus 16 
percent). 

While we cannot say what would have happened to the teacher 
researcher participants had they attended teacher development institutes 
(and statements about relative gain cannot be derived from the study), 
it is clear from Figure 2 that overall, the research participants gained 
skills and knowledge that were associated with changes in classroom 
practice. This suggests that changes in practice can be attained in a 
variety of ways. The model classroom may be a very effective, but not 
the only effective, way to promote changes in classroom instruction. 

More attention needs to be given to developing an understanding of 
the benefits of different strategies for different teachers or for teachers 
at different points in their careers. 

Teacher Leadership and Dissemination of Findings 

A commonly held belief is that teachers who participate in summer 
institutes and other extended professional development efforts will not 
only change their own classroom practices, but they also will work 
with other teachers in their schools or districts to bring about more 
widespread change. The belief, or hope, is that by investing resources 
in the training of one or two teachers, many more teachers will be 
reached. 

Findings from the study suggest, however, that this belief is not 
readily substantiated. Granted, teacher leadership was not an explicit 
goal for the majority of programs, and strategies for dissemination j 

were covered lightly, if at all. However, many of the programs cited j 

i 

i 



o 

ERIC 



21 



Findings from the Multi- Agency Study @3 



dissemination as a desired benefit, and dissemination of new practices 
is frequently mentioned as a general goal of professional development 
activities. . ; 

Both the case stories arid survey data strongly indicate that formal 
dissemination of knowledge and practice was a relatively rare event. 
Figures 3 and 4 present responses from the teacher development 
and teacher researcher participants regarding , their d issemination 
activities. These data show that in general, little direct dissemination 
of knowledge and skills resulted from prograrri exposure. 

However, in our program where leadership was a major focus, a 
somewhat more positive picture emerged. The case story found: 

All the teachers interviewed, with the exception of one who 
worked primarily alone, had shared Mai they had learned in 
the program with others. They had done so through holding 
demonstrations in their own classrooms for other teachers to 
observe, conducting workshops for their colleagues at different 
times, guiding teachers through FOSS, kits, and working 
closely with the program’s mathematics representatives or 
science representatives at different grade levels. Most of 
their activities were confined to their own buildings unless 
they were teacher leaders who had responsibilities at other 
schools. Their formal contacts with teacher-participants in 
schools other than their own were limited primarily to their 
regular institute sessions. The amount of sharing varies by 
building. 

This relative lack of leadership in dissemination among the majority 
of participants is especially noteworthy as a number of other indicators 
suggest that the teachers who participate in summer institutes are 
among the more outgoing and may already have held leadership 
positions in their schools or districts. While one reason may be the 
lack of formal emphasis on dissemination in most of the programs, 
another is probably the lack of time and/or support when the teachers 
return to their schools. This latter issue, and the overall role of the 
school or district context, is discussed in more detail below. 

The Importance of Supports at the District and School Levels 

Many people strongly believe that the most effective strategies for 
professional development tie the development directly to the context 
of a particular district or school and rely on the teachers themselves for 
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Percent of participants in teacher development programs saying they 
have shared and disseminated knowledge gained in the program to a 
great extent 
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Figure 4 

Percent of research participants saying that they have shared and 
disseminated knowledge gained in the program to a great extent 



planning the program and designing (or in some cases providing) its content 
(Lieberman, 1995; Darling-Hammond, 1995). These researchers advocate 
models that tie professional development to a particular school and are 
explicitly linked to reform activities that the school is undertaking. 

At the time that the multi-agency study was initiated, the emphasis on 
the individual school-based model was not as strong as it is today. Rather, 
the emphasis was on ensuring that professional development efforts were 
designed as sustained and extended experiences that would be supported 
by school or district personnel. 10 Ties might be explicit in terms of 



10 This is not to say that teacher input was unimportant. However, this input was sought 
more frequently in terms of evaluation and feedback rather than planning and direction. 
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integration with systemic reform efforts or implied through loosely 
structured agreements with school authorities. For example, many 
programs required that a teacher’s principal sign a letter agreeing to 
have the teacher participate as a condition of enrollment. Underlying 
this requirement was the belief (stated or unstated) that this signature 
served as a proxy for future support to the teacher in using and sharing 
new skills after program participation. 

The present study affirms the importance of support at the 
school or district level and was critical in assuring application of the 
lessons of the professional development experience. The impact of 
the professional development experience — translation to classroom 
instruction, leadership, dissemination, etc. — was strongly affected by 
the district or school context already in place. As stated earlier, more 
translation to the classroom was found where teachers felt they had 
more control over their schools and classrooms. The case stories also 
suggest that teachers with a supportive administrator are able to make 
the most changes. Some contrasting examples are found in one of the 
case stories. 

Bob considers his principal to be supportive. He said that the 
principal seems to have an interest in science. If it 's within 
reason, the principal will give Bob release time to attend a 
function and will spend money for the classroom.... 

Two participants indicated that they receive much 
psychological support but little financial support for 
computers, other equipment and supplies.... Another teacher 
receives no support from his department chair. He was 
“called on the carpet" by the principal for not having his 
students use the textbook. 

Another important factor affecting classroom implementation is 
the availability of materials and equipment. It was clear that many 
of the new skills and techniques learned at the institutes could not 
be implemented, or were far more difficult to implement in situations 
where needed materials and equipment were not available. For 
example, a case story that focused on the impacts of an institute aimed 
at enhancing teachers’ ability to use computers found the following: 

The greatest barriers that teachers perceive in their classroom 
implementation of [a program focusing on computer usage] 
are ...no money to buy updated computers; lack of access to 
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•t. computers in the classroom; and lack of available software 

relevant to the content areas being taught. 

Recognizing this factor, some programs have built materials provision 
into their programs. In some cases, low-cost, easy-to-access materials 
are provided to teachers in the “make-and-take” mode. In other cases, 
traveling vans bring high-end equipment directly to the schools to use 
for' a limited period of time. 

However, material and equipment supports are not the only, or 
even the most important, factor in determining whether changes in 
practice take place. A more important facilitator is the general reform 
climate in the school or district and the “readiness for change.” The 
survey data showed that in addition to teachers’ previous practices, 
level of school taught, and sense of autonomy, a major facilitator of 
transfer was teaching in a school in which there was already some 
kind of reform being implemented. 

These findings, taken together, reinforce the importance of 
professional development programs having more than a superficial 
tie to the school and highlight the need for a strong commitment of 
support if the goal of improved classroom learning is to be widely 
attained. To a large extent, the findings also lend credence to the 
importance of school-based programs and suggest that such programs 
may provide the most immediate effect on changing practices. At its 
best, the school-based model brings with it a number of characteristics 
shown by this study to be needed for change to occur — a readiness 
for change, a supportive administrative structure, a cadre of peers who 
can support each other, and time for trying out new approaches and 
introducing new content — all critical supports for change. 

The results also indicate that the school- or district-based model 
is not the only one that can be effective. Many participants in 
professional development programs with a regional and even national 
target population were able to make significant changes in their 
classroom practice. What seems to be most important is making sure 
that certain predisposing conditions are established in the school or 
district — a significant challenge in far too many instances. 

Conclusions 

Taken together, the results from this multi-agency study provide 
some valuable insights into the effects of participating in professional 
development programs. First, the findings confirm the value of 
well-designed professional development experiences for those who 
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attend. Not only were they seen by participants to be personally and 
professionally valuable, but the data show that a teacher’s participation 
has important effects on what happens in his/her classroom. Although 
we cannot say that these experiences result in increased student 
achievement, we do know that they contribute to establishing the 
conditions under which improved achievement is likely to occur. 
Programs that provide teachers increased knowledge about, and 
practice in, delivering standards-based science instruction result in 
students being provided pedagogy and broader exposure to relevant 
and important science content. 

Even among this self-selected group of attendees, however, 
there were some important differences in outcomes. Some practices 
associated with standards-based instruction were evidenced far less 
frequently than others. For example, it appears difficult for teachers 
to change from limited coverage of a wide range of topics to more 
indepth consideration of a few topics. Some teachers also are resistant 
to giving up a teacher-centered approach. And, while hands-on, 
instruction was the most widely observed practice, there were still 
indications that more progress could be made in assuring that hands-on 
really means minds-on, rather than a more superficial adoption of ! 
active engagement. 

Second, programs that model the ideal science classroom, 
including participating in hands-on activities for use in the classroom, 
planning how information could be used in the classroom, developing j 
curriculum units, engaging in challenging problem solving, and j 
collaborating with scientists or other staff, are the most effective ip j 
facilitating this transfer. While we cannot be certain that this.approach i 

is preferable in the long run, in the short term programs tied directly to > 

what is desired in the.classroom have the most immediate payoff. 

third, the school or district context, and the support provided : 
for the application of learning, make a big difference in the 
extent to which changes actually occur. While it is certainly 
possible for the lone teacher, unsupported by administrators and 
colleagues, to make a difference, the chances of this occurring are 
far less than in a supportive environment. This finding raises some 
possible dilemmas. Specifically, should teachers be denied access 
to professional development if the school or district commitment 
for support is not adequate? And, what are the characteristics of an 
adequate commitment? Clearly, a signature on a letter of application is 
not. Materials and equipment support, release time to attend followup 
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sessions, and work with other teachers are more desirable. Having 
teams of teachers participate is also a potentially effective strategy. 
These, plus an explicit tie-in to an ongoing plan for reform, are 
probably the best. The very difficult question is how far program 
managers should go in defining and requiring commitment from home 
schools and districts. 

Fourth, the question of moving the high school teacher toward 
greater reform remains a significant challenge. The good news is that 
participants who teach at the high school level do make changes; the 
bad news is that these changes are significantly smaller in magnitude. 
This is not the first study to note that it is more difficult to change the 
behavior of high school teachers than teachers in the earlier grades 
(Cuban & Tyack, 1 995). However, how to change this phenomenon 
remains an open question, and a closer look at the relative efficacy of 
programs that target this population seems warranted. 

Finally, those who are interested in using professional development 
programs as a way to promote teacher leadership and dissemination 
of program benefits need to recognize that the link between program 
participation and sharing of information is not necessarily given or 
naturally evolving. A variety of factors affect whether or not a teacher 
is able to share what has been learned with others. While this is no 
doubt related to the environment of the school, there are a number 
of ways in which the professional development program can increase 
the likelihood that such sharing occurs. First, teachers must be made 
aware that it is an expectation of their program participation. Second, 
administrators in the schools where participants teach need to make 
a commitment that they will not only encourage their leadership in 
information dissemination, but will also provide time and support for 
it to occur. And, third, professional development programs need to 
include activities specially designed to build the capacity of teachers 
to become leaders in the educational change process, helping them to 
both identify strategies for playing more visible roles and giving the 
skills needed to promote the acquisition of the new understandings 
among their peers and colleagues. 
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